Optimal Filters for Gradient-based Motion Estimation

Gradient based approaches for motion estimation Optical Flow estimate the motion of an image sequence based on local changes in the image intensities In order to best evaluate local changes in the intensities speci c lters are applied to the image sequence These lters are typically composed of spatio temporal deriva tives The design of these lters plays an important role in the estimation accuracy This paper proposes a method for the design of these lters in an optimal manner Unlike previous approaches that design optimal derivative lters in some sense the proposed technique de nes the optimality directly with respect to the motion estimation goal The suggested approach takes into account prior knowledge on the motion distribution the image characteristics and the allocated lter length Simulations demonstrate the advantage of the new design approach Introduction Estimating motion between two images plays a vital role in many applications and has drawn a lot of attention during the last two decades There are many ways to approach this problem and indeed many algorithms have been proposed for this task e g In Barron et al a comparative survey of many motion estimation techniques is given One family of such algorithms which was found to perform well is the family of gradient based methods originally proposed by Horn and Schunck The gradient based methods emerge from the assumption that the intensity value of a physical point in a scene does not change along the image sequence Denoting the intensity values of the image sequence by the function I x y t where x y is the spatial position and t is the temporal axis the brightness constancy assumption along the image stream yields dI x y t dt I x dx dt I y dy dt I t De ning u u dx dt dy dt as the spatial velocity of each spatio temporal point in the image sequence we obtain Ixu x Iyu y It Here Ix Iy and It denote the spatial and temporal derivatives This Brightness Constancy Equation BCE relates the spatial and temporal gradients of an image sequence to the motion vector u u at each location x y t Since the above equation forms a single constraint over the two component motion vector more constrains are required to uniquely recover the motion eld For this purpose an assumption of smoothness spatial and or temporal is typically imposed One issue that is critical to the implementation of the above BCE is that image derivatives are computed based on sampled information It is commonly agreed that approximating the spatio temporal derivatives by nite di erences produces error in the above equation and subsequently in the estimated motion One of the major conclusions of Barron is that the method of numerical di erentiation is very important di erences between rst order pixel di erencing and higher order central di erences are very noticeable In most implementations spatio temporal smoothing is applied to the image sequence prior to motion estimating Since nite gradients are more accurate at low frequencies pre smoothing attenuates spatial and temporal aliasing e ects and improves the overall accuracy of gradient estimation Pre smoothing and gradient operations are both Linear and Spatio Temporal Invariant LSTI Therefore it is possible to combine them into a single ltering operation In the most general case the BCE can be implemented in the following way Ixu x Iyu y It fF Igu x fF Igu y fF Ig where F F and F are spatio temporal digital lters of some sort and fA Bg denotes discrete convolution operation between two D signals Several attempts to de ne or design these lters together or separately have been reported in the literature All these methods treat the above question as a problem of optimally designing gradient operators overlooking the fact that these gradients are to be used for motion estimation The question addressed in this paper is that of designing these lters such that they are optimal with respect to the motion estimation goal Since D separable lters are easier to implement it is commonly demanded that F F and F are separable We adopt this line of reasoning in this paper as well Existing Motion Estimation Filters The numerical analysis literature contains many methods for approximating gradient lters Most of the papers describing optical ow estimation using the BCE apply simple gradient lters such as e g In many papers the choice of these lters is not even mentioned In their original paper Horn and Schunck proposed an approximation of the gradient lter with no pre smoothing The gradients were obtained by averaging the rst di erences over a cube of pixels in the image sequence These gradients refer to a center point of the cube which means that the estimated ow corresponds to points between pixels No motivation or justi cation for this choice of gradient estimation is given According to Barron et al these gradient lters are said to be a relatively crude form of numerical di erentiation and can be the source of considerable error Barron et al propose the application of a spatio temporal pre smoother constructed using a sampled Gaussian lter with variance at each axis This variance was found empirically to give the best results The gradient lter proposed by Barron is the tap central di erence lter which is the result of a design procedure described in In this scheme the goal is to obtain a near accurate gradient transfer function D j where the lter coe cients fd k gLk are designed to meet this requirement as closely as possible The derived D gradient lter is used to produce types of derivatives x derivative y derivative and t derivative in a separable manner a D pre smoothing kernel is rst applied to the image sequence Then each axis is di erentiated separately applying the obtained derivative lter Figure upper graph depicts the power spectrum of the gradient lter j D j jD S j compared to an analytic di erentiation of the smoothing lter jj S j The pre smoother is taken to be a sampled Gaussian lter with a variance of and the gradient lter is the taps central di erence lter as suggested by Barron et al It is demonstrated that the error between these two responses is very small for low frequencies but increases as the frequency tends to In Simoncelli proposed that the pre smoother and the derivative lters should be well matched that is the lter d x should be the rst derivative of the lter s x For digital lters this requirement is stated with respect to some choice of interpolant If we denote a smoothing and a derivating D lter pair in the frequency domain by S and D respectively then the error j S D can be minimized in a more accurate manner For example high frequencies which are not treated correctly by D can be attenuated by the pre smoother S in order to minimize the above approximation error Figure lower graph shows the frequency response of the gradient lter D compared to an analytic gradient of the smoother lter j S This time the error between these two is negligible Existing Approaches Is It Really The Best We Can Do The existing methods for designing lters to be used in optical ow estimation aim at obtaining lters which are as similar as possible to derivatives However all existing methods over look the nal goal of these lters namely the estimation of optical ow In this paper we rst propose a technique to derive a set of lters which are optimal speci cally with respect to this goal These lters are designed to give the best estimation results in term of accuracy where their derivative characteristics are not a demand but a by product In our scheme we adopt useful design requirements from existing methods and add a few more −4 −3 −2 −1 0 1 2 3 4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Frequency −4 −3 −2 −1 0 1 2 3 4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5